{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transformers installation\n",
    "! pip install transformers datasets evaluate accelerate\n",
    "# To install from source instead of the last release, comment the command above and uncomment the following one.\n",
    "# ! pip install git+https://github.com/huggingface/transformers.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Knowledge Distillation for Computer Vision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knowledge distillation is a technique used to transfer knowledge from a larger, more complex model (teacher) to a smaller, simpler model (student). To distill knowledge from one model to another, we take a pre-trained teacher model trained on a certain task (image classification for this case) and randomly initialize a student model to be trained on image classification. Next, we train the student model to minimize the difference between its outputs and the teacher's outputs, thus making it mimic the behavior. It was first introduced in [Distilling the Knowledge in a Neural Network by Hinton et al](https://huggingface.co/papers/1503.02531). In this guide, we will do task-specific knowledge distillation. We will use the [beans dataset](https://huggingface.co/datasets/beans) for this.\n",
    "\n",
    "This guide demonstrates how you can distill a [fine-tuned ViT model](https://huggingface.co/merve/vit-mobilenet-beans-224) (teacher model) to a [MobileNet](https://huggingface.co/google/mobilenet_v2_1.4_224) (student model) using the [Trainer API](https://huggingface.co/docs/transformers/en/main_classes/trainer#trainer) of 🤗 Transformers.\n",
    "\n",
    "Let's install the libraries needed for distillation and evaluating the process.\n",
    "\n",
    "```bash\n",
    "pip install transformers datasets accelerate tensorboard evaluate --upgrade\n",
    "```\n",
    "\n",
    "In this example, we are using the `merve/beans-vit-224` model as teacher model. It's an image classification model, based on `google/vit-base-patch16-224-in21k` fine-tuned on beans dataset. We will distill this model to a randomly initialized MobileNetV2.\n",
    "\n",
    "We will now load the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dataset = load_dataset(\"beans\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use an image processor from either of the models, as in this case they return the same output with same resolution. We will use the `map()` method of `dataset` to apply the preprocessing to every split of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoImageProcessor\n",
    "teacher_processor = AutoImageProcessor.from_pretrained(\"merve/beans-vit-224\")\n",
    "\n",
    "def process(examples):\n",
    "    processed_inputs = teacher_processor(examples[\"image\"])\n",
    "    return processed_inputs\n",
    "\n",
    "processed_datasets = dataset.map(process, batched=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Essentially, we want the student model (a randomly initialized MobileNet) to mimic the teacher model (fine-tuned vision transformer). To achieve this, we first get the logits output from the teacher and the student. Then, we divide each of them by the parameter `temperature` which controls the importance of each soft target. A parameter called `lambda` weighs the importance of the distillation loss. In this example, we will use `temperature=5` and `lambda=0.5`. We will use the Kullback-Leibler Divergence loss to compute the divergence between the student and teacher. Given two data P and Q, KL Divergence explains how much extra information we need to represent P using Q. If two are identical, their KL divergence is zero, as there's no other information needed to explain P from Q. Thus, in the context of knowledge distillation, KL divergence is useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments, Trainer\n",
    "from accelerate import Accelerator\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class ImageDistilTrainer(Trainer):\n",
    "    def __init__(self, teacher_model=None, student_model=None, temperature=None, lambda_param=None,  *args, **kwargs):\n",
    "        super().__init__(model=student_model, *args, **kwargs)\n",
    "        self.teacher = teacher_model\n",
    "        self.student = student_model\n",
    "        self.loss_function = nn.KLDivLoss(reduction=\"batchmean\")\n",
    "        device = Accelerator().device\n",
    "        self.teacher.to(device)\n",
    "        self.teacher.eval()\n",
    "        self.temperature = temperature\n",
    "        self.lambda_param = lambda_param\n",
    "\n",
    "    def compute_loss(self, student, inputs, return_outputs=False):\n",
    "        student_output = self.student(**inputs)\n",
    "\n",
    "        with torch.no_grad():\n",
    "          teacher_output = self.teacher(**inputs)\n",
    "\n",
    "        # Compute soft targets for teacher and student\n",
    "        soft_teacher = F.softmax(teacher_output.logits / self.temperature, dim=-1)\n",
    "        soft_student = F.log_softmax(student_output.logits / self.temperature, dim=-1)\n",
    "\n",
    "        # Compute the loss\n",
    "        distillation_loss = self.loss_function(soft_student, soft_teacher) * (self.temperature ** 2)\n",
    "\n",
    "        # Compute the true label loss\n",
    "        student_target_loss = student_output.loss\n",
    "\n",
    "        # Calculate final loss\n",
    "        loss = (1. - self.lambda_param) * student_target_loss + self.lambda_param * distillation_loss\n",
    "        return (loss, student_output) if return_outputs else loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now login to Hugging Face Hub so we can push our model to the Hugging Face Hub through the `Trainer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login\n",
    "\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's set the `TrainingArguments`, the teacher model and the student model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForImageClassification, MobileNetV2Config, MobileNetV2ForImageClassification\n",
    "\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=\"my-awesome-model\",\n",
    "    num_train_epochs=30,\n",
    "    fp16=True,\n",
    "    logging_strategy=\"epoch\",\n",
    "    eval_strategy=\"epoch\",\n",
    "    save_strategy=\"epoch\",\n",
    "    load_best_model_at_end=True,\n",
    "    metric_for_best_model=\"accuracy\",\n",
    "    report_to=\"tensorboard\",\n",
    "    push_to_hub=True,\n",
    "    hub_strategy=\"every_save\",\n",
    "    hub_model_id=repo_name,\n",
    "    )\n",
    "\n",
    "num_labels = len(processed_datasets[\"train\"].features[\"labels\"].names)\n",
    "\n",
    "# initialize models\n",
    "teacher_model = AutoModelForImageClassification.from_pretrained(\n",
    "    \"merve/beans-vit-224\",\n",
    "    num_labels=num_labels,\n",
    "    ignore_mismatched_sizes=True\n",
    ")\n",
    "\n",
    "# training MobileNetV2 from scratch\n",
    "student_config = MobileNetV2Config()\n",
    "student_config.num_labels = num_labels\n",
    "student_model = MobileNetV2ForImageClassification(student_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `compute_metrics` function to evaluate our model on the test set. This function will be used during the training process to compute the `accuracy` & `f1` of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import evaluate\n",
    "import numpy as np\n",
    "\n",
    "accuracy = evaluate.load(\"accuracy\")\n",
    "\n",
    "def compute_metrics(eval_pred):\n",
    "    predictions, labels = eval_pred\n",
    "    acc = accuracy.compute(references=labels, predictions=np.argmax(predictions, axis=1))\n",
    "    return {\"accuracy\": acc[\"accuracy\"]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's initialize the `Trainer` with the training arguments we defined. We will also initialize our data collator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DefaultDataCollator\n",
    "\n",
    "data_collator = DefaultDataCollator()\n",
    "trainer = ImageDistilTrainer(\n",
    "    student_model=student_model,\n",
    "    teacher_model=teacher_model,\n",
    "    training_args=training_args,\n",
    "    train_dataset=processed_datasets[\"train\"],\n",
    "    eval_dataset=processed_datasets[\"validation\"],\n",
    "    data_collator=data_collator,\n",
    "    processing_class=teacher_processor,\n",
    "    compute_metrics=compute_metrics,\n",
    "    temperature=5,\n",
    "    lambda_param=0.5\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now train our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate the model on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.evaluate(processed_datasets[\"test\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On test set, our model reaches 72 percent accuracy. To have a sanity check over efficiency of distillation, we also trained MobileNet on the beans dataset from scratch with the same hyperparameters and observed 63 percent accuracy on the test set. We invite the readers to try different pre-trained teacher models, student architectures, distillation parameters and report their findings. The training logs and checkpoints for distilled model can be found in [this repository](https://huggingface.co/merve/vit-mobilenet-beans-224), and MobileNetV2 trained from scratch can be found in this [repository](https://huggingface.co/merve/resnet-mobilenet-beans-5)."
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 4
}
